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ERROR CONCEALMENT OF VIDEO DATA USING 



TEXTURE DATA RECOVERY 



BACKGROUND 



The present invention relates to error concealment, and 
more particularly, to concealing errors in texture partition 
of a video packet . 

In an MPEG-4 decoder used for real-time streaming, when 
an error is detected in the texture partition of a video 
packet, substantially all decoded texture information of that 
packet is often discarded. In video data, texture represents 
luminance and chrominance information of the pixels in the 
frame. The region of the video frame corresponding to this 
corrupted packet may then be generated by error correction 
process. However, traditional error detection and correction 
systems may require a significant amount of overhead as well 
as significant amount of data processing when decoding coded 
video bit stream signals. Thus, current video decoding 
systems may rely on error concealment techniques for 
effective video streaming. In contrast to error correction, 
which attempts to reconstruct lost or corrupt data, error 
concealment aims to generate data that may be substituted for 
the lost or corrupt data. Discrepancies in image created by 
the generated data (generally at the macroblock level) are 
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not likely to be perceived by a viewer of a video image that 
utilizes such error concealment. 

Motion compensated temporal replacement may be an 
appropriate error concealment technique in video frames 
without much scene changes. This technique uses motion 
information to propagate macroblocks of the texture data from 
the" previous frame to replace the texture data of the 
corrupted macroblocks in the current frame. Thus, this 
technique may provide effective error concealment as long as 
motion information can be obtained without much error. 
However, there are times when the texture data cannot be 
adequately concealed just by copying texture data from the 
previous frame at some motion compensated displacement. 
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SUMMARY 

In one aspect, a method for concealing errors in texture 
partition of a video packet is described. The method 
includes determining a particular macroblock within the 
texture partition where error is detected, concealing the 
error starting at the particular macroblock, and evaluating 
image smoothness of concealed macroblocks . The concealing 
and evaluating are repeated with one more macroblock added 
prior to the previous particular macroblock. The concealing 
and evaluating are repeated until all macroblocks in the 
texture partition have been concealed. A set of macroblocks, 
including a combination of decoded and concealed macroblocks, 
that produces best image smoothness is then selected to 
replace the corrupted texture partition. 

In another aspect, an error concealment system for 
texture partition of a video packet is disclosed. The system 
includes an error location detector, an error concealment 
element, an image smoothness evaluator, and a selector. The 
error location detector receives video packets, and 
determines a particular macroblock within the texture 
partition where error is detected. The error concealment 
element conceals the error starting at the particular 
macroblock. The image smoothness evaluator evaluates the 
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concealed macroblocks . The selector then selects a set of 
macroblocks , including a combination of decoded and concealed 
macroblocks, that produces best image smoothness . 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figures 1A through IB describe the present error 
concealment technique in accordance with an embodiment of the 
present invention . 

Figure 2 shows an example of a corrupted video packet 
with relevant parameters designated on the packet. 

Figure 3 illustrates an image smoothness test of texture 
data in accordance with an embodiment of the present 
invention . 

Figure 4 shows an error concealment system according to 
an embodiment of the present invention. 
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DETAILED DESCRIPTION 



In recognition of the above-stated difficulties with the 
prior error concealment techniques, the present invention 
describes embodiments for error concealment of video data 
5 using texture data recovery. In concealing corrupted video 
packets, the decoder utilizes uncorrupted received texture 
information. Macroblock boundary smoothness is evaluated for 
varying number of macroblocks of utilized texture 
information. The final number of macroblocks of utilized 

□ 

*Q 10 decoded texture information (the recovered information) is 
iij selected to maximize overall boundary smoothness. Although 

IV 

H present embodiments are applicable to any video data, they 

Sj are particularly well suited for MPEG- 4 decoders used for 

e: 

O real-time streaming video. Consequently for purposes of 

I s * 15 illustration and not for purposes of limitation, the 

m 

P exemplary embodiments of the invention are described in a 

H 

manner consistent with such use, though clearly the invention 
is not so limited. 



FIGS . 1A through IB describe the present error 
20 concealment technique in accordance with an embodiment of the 
present invention. The technique is described in a flowchart 
form. 
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In the illustrated embodiment, determination is made (at 
100) as to the presence of any errors in the texture 
partition of a video packet. Detecting invalid variable 
length code (VLC) , inconsistent resynchronizat ion header 
information, and/or receipt of out -of -range motion vectors 
and other related invalid codes may determine video packet 
errors . Since variable length codes are used to compress 
video data, the location where the decoder detects an error 
(e.g. where invalid VLC occur) may be some undetermined 
number of bits away from the actual location of the error. 
Accordingly, if the error is detected at 102, then the video 
packet may be further processed at 106. Otherwise if no 
error is detected at 102, then the next video packet may be 
processed at 104. FIG. 2 shows an example of a corrupted 
video packet 200 . 

In video data encoded in Moving Picture Experts Group 
(MPEG) format, a video frame is divided into macroblocks . In 
this format, each video packet includes information for 
consecutive macroblocks from left to right and top to bottom. 
The number of macroblocks {N) represented by each video 
packet may be determined by the change in macroblock number 
field of the video packet header. Thus, packet processing 
(at 106) may include determining the location of the detected 
error 202 with respect to the total number of macroblocks (N) 
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204. This error location determination may be made as 
occurring at a particular macroblock (JC) 206 from the end of 
the texture partition 210. Alternatively, the error location 
may be determined to occur at N-K macroblocks from the 
beginning of the texture partition 208. Therefore, an 
assumption may be made that the texture information is 
corrupted starting from the location of the detected error 
202. Accordingly, K is initialized to the number of 
macroblocks in the packet that was decoded after the first 
detected error of the packet. This initial value of K (i.e. 
Kinit) 206 is the least number of macroblocks that is 
concealed. 

Since N-K in i t is the largest number of macroblocks to be 
decoded but not concealed, the texture information of the 
first N-Ki n i t decoded macroblocks is stored in a temporary 
buffer, at 108, to be used later in selecting a set of 
texture data with the best boundary smoothness measure. 
Further, the last K macroblocks of the texture data is error 
concealed, at 110. In the illustrated embodiment of FIG. 1A, 
the error concealment process, at 110, includes motion 
compensated temporal replacement. This may involve 
concealing the corrupted texture data with motion compensated 
texture data from the previous frame using corresponding 
motion vectors. To save computation time, motion compensated 
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temporal replacement may only be required on macroblocks 
whose motion vector has changed. The first N-K macroblocks 
assumed to have non-corrupted texture information may be 
considered as "non-corrupted" in the above error concealment 
process. 

In the illustrated embodiment, the image smoothness of 
the corrupted video packet macroblocks restored by temporal 
replacement (at 110) may then be evaluated, at 112. The 
image smoothness may be measured by computing the smoothness 
of macroblock boundaries in the restored texture data. In 
one embodiment, this boundary smoothness is measured by 
summing the pixel value mismatch between macroblock boundary 
pixels. Thus, this method measures the image smoothness 
spatially (see FIG. 3) . This embodiment is described in 
detail below in conjunction with description of FIG. 3. 

The value of K that produces the best overall boundary 
matching (i.e. smallest sum of boundary pixel mismatch for 
all macroblocks of the corrupted video packet) is stored, at 
114, as K bes t for later use. Moreover, by storing the partial 
sum of the mismatch values (mismatch values for each 
macroblock) , the sum of the mismatch values in each iteration 
may be computed more efficiently by reusing the partial 
mismatch values from previous iteration for macroblocks that 
itself and its neighbors did not change. 
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At 116, the variable K is incremented by one, which 
moves the particular point of demarcation between the decoded 
and the concealed texture data further to the beginning 208 
of the texture partition in the video packet. If this 
particular point of demarcation has not reached the beginning 
of the motion partition 206 in the video packet (i.e. K^N) , 
at 118, then the process of error concealing by creating 
motion compensated temporal replacement and evaluating the 
image smoothness /matching with one more macroblock of 
concealed texture data is repeated, starting at 110. 
Otherwise when K = N, all macroblocks of the texture data 
have been concealed. Thus when K - N, a combination of 
recovered decoded texture data and concealed texture data 
that produced a lowest mismatch value (i.e. best image 
smoothness at K = K bGSt ) is selected, at 120. Accordingly, 
the texture data of the first N-K best macroblocks is restored 
from the temporary buffer created in 108. The texture data 
of these N~K best macroblocks is the recovered texture data. 
The last Kbest macroblocks then include concealed texture 
data. Finally, the processing moves to the next video 
packet , at 122 . 

FIG. 3 illustrates an image smoothness test of texture 
data in accordance with an embodiment of the present 
invention. The image smoothness of the current macroblock 
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300 may be evaluated by summing the square of the pixel value 
mismatch between the current macroblock 3 00 and the 
neighboring macroblock 302, 304, 306, 308 boundary pixels. 
However, other methods, such as root -sum- square or simple 
difference sum, may be employed to compute the boundary pixel 
mismatch value. Further, the image smoothness test may be 
performed with more than one row or column difference of the 
boundary pixels between neighboring macroblocks . 

In the illustrated embodiment, the boundary pixel 
mismatch value (jxl) of the current macroblock 3 00 is computed 
as : 

15 
i=0 

The weights w a/ w b , w X/ and w r may be used to weigh the 
boundary mismatch between macroblocks belonging to different 
video packets differently. Thus, depending on the desired 
quality of video frames, the boundary mismatch between 
macroblocks that belong to different video packets may be 
configured to weigh more than the mismatch between the 
macroblocks that belong to same video packets. 

An error concealment system 400 according to an 
embodiment of the present invention is shown in FIG. 4. The 
system 400 includes an error location detector 402 to receive 
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N macroblocks of texture data, some of which are corrupted. 
In the illustrated embodiment, the detector 402 is configured 
to determine the location of a video packet error with 
respect to the total number of macroblocks (N) . The location 
5 determination provides the macroblock number (K ini t) from the 
end 210 of the texture partition. Thus, this determination 
enables separation of N macroblocks of texture data into N-K 
recovered macroblocks and K concealed macroblocks. 

The N-K± nit macroblocks are stored in a storage element 
q 10 404. The first N-K besC macroblocks of the N-K in i t macroblocks 
ijg are later used, once the value for K best is determined. The 

u 

fly last K init macroblocks of the texture data are then sent to an 

E-JL 
5 — 

fy error concealment element 406. In one embodiment, the error 

ji concealment element 406 includes a motion compensated 

O 15 temporal replacement element . The temporal replacement 
~ « 

ifl element uses motion information to propagate macroblocks of 

o 

jO: the texture data from the previous frame to replace the 

texture data of the macroblocks in the current frame. The 
first N-K macroblocks assumed to have non-corrupted texture 
20 information may be considered as "non-corrupted" in the above 
temporal replacement process . 

An image smoothness evaluator 408 then evaluates the 
image smoothness of the temporally replaced texture data. In 
the illustrated embodiment, the evaluator 408 measures the 

12 
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smoothness of macroblock boundaries in the restored texture 
data. Particular implementations of the macroblock boundary 
test have been described above. 

A best image smoothness selector 410 receives a pixel 
mismatch value from the evaluator 408, and selects texture 
data that provides best image smoothness. This selection 
provides a value for K best . The selector 410 provides this 
value to the error concealment element 406 and the storage 
element 404. In response, the storage element 404 provides 
N-Kb est recovered decoded macroblocks, and the error 
concealment element 406 provides K best concealed macroblocks . 
These macroblocks of texture data may be combined to produce 
a best set of macroblocks of texture data to replace the 
corrupted video packet . 

There has been disclosed herein embodiments for error 
concealment of video data using texture data recovery. In 
one embodiment, the corrupted macroblocks may be concealed by 
motion compensated temporal replacement. To find an 
appropriate number of macroblocks to conceal, macroblock 
boundary smoothness is evaluated for varying number of 
macroblocks of utilized texture information. Thus, the final 
number of macroblocks of utilized decoded texture information 
(the recovered information) is selected to maximize overall 
boundary smoothness. 

13 
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While specific embodiments of the invention have been 
illustrated and described, such descriptions have been for 
purposes of illustration only and not by way of limitation. 
Accordingly, throughout this detailed description, for the 
purposes of explanation, numerous specific details were set 
forth in order to provide a thorough understanding of the 
present invention. It will be apparent, however, to one 
skilled in the art that the system and method may be 
practiced without some of these specific details. For 
example, error concealment may use a process other than the 
motion compensated temporal replacement. Moreover, the 
present embodiments may be used to process video data having 
a texture data unit other than a macroblock. Further, the 
methods of the present embodiments may be programmed as a 
software on a computer readable medium, such a floppy disk or 
a CD-ROM disk. In other instances, well-known structures and 
functions were not described in elaborate detail in order to 
avoid obscuring the subject matter of the present invention. 
Accordingly, the scope and spirit of the invention should be 
judged in terms of the claims which follow. 
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